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ABSTRACT 



This paper describes implementation of one private 
university's strategic plan which includes provision of laptop computers to 
all entering freshmen and faculty since fall 1996. The background and context 
of the development of this plan is summarized, as are the 10 assessment 
instruments, both standard and in-house, being used to measure the 
effectiveness of the entire plan. The computing component has involved hiring 
of 17 academic computer specialists to provide faculty support, revamping the 
school's Information Systems Support Center, and establishment by faculty of 
a special initiative to develop effective uses of computers in instruction. 
Preliminary evaluation findings indicate: significantly higher usage of 
computers by students, significantly higher computer skills among faculty, 
and improved attitudes toward computers. Among academic outcomes, there is 
evidence that the computing initiative has had a positive impact with the 
retention rate of first -year students higher than in the past. However, in 
non- computer areas, there was some evidence that the initiative might have 
impacted students so that they may be less rounded than previous students . 
(Contains 11 references.) (DB) 



***************************************************************************** 

* Reproductions supplied by EDRS are the best that can be made 

* from the original document. 




VO 



CO 

CO 



Assessment of the Impact of Ubiquitous Computing 



Director of Institutional Research and Academic Administration 

Wake Forest University 
Box 7373 Reynolda Station 
Winston-Salem, NC 27109 
336 758-5020 



Yihong Gu 

Associate Director of Institutional Research 
Wake Forest University 
Box 7373 Reynolda Station 
Winston-Salem, NC 27109 
336 758-5161 



David G. Brown 

Vice President and Dean 

International Center for Computer Enhanced Learning (ICCEL) 

Wake Forest University 
Box 7328 Reynolda Station 
Winston-Salem, NC 27109 
336 758-4878 



Ross A. Griffith 



BEST COPY AVAILABLE 



PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL HAS 
BEEN GRANTED BY 



/ CENTER (ERIC) 

fcr This document has been reproduced as 
received from the person or organization 
originating it. 






U.S. DEPARTMENT OF EDUCATION 
Office of Educational Research and Improvement 
EDUCATIONAL RESOURCES INFORMATION 



3 

oo 



D. Vura 



2 




□ Minor changes have been made to 
improve reproduction quality. 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 



• Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. 



for Management Research, Policy Analysis, and Planning 



This paper was presented at the Thirty-Ninth Annual Forum 
of the Association for Institutional Research held in Seattle, 
Washington, May 30-June 3, 1999. 

This paper was reviewed by the AIR Forum Publications 
Committee and was judged to be of high quality and of 
interest to others concerned with the research of higher 
education. It has therefore been selected to be included in 
the ERIC Collection of AIR Forum Papers. 



Dolores Vura 
Editor 

Air Forum Publications 




3 



Assessment of the Impact of Ubiquitous Computing 



Paper Abstract - A Doctoral II private university is in the final stages 
of implementing a novel strategic plan that has received considerable 
international attention. The most distinctive outcome of the plan is that all 
faculty and entering freshmen as of 1996 received IBM laptop computers. 
Articles about this program of ubiquitous computing have been published in 
the Chronicle of Higher Education and various other publications. 
Extraordinary efforts to assess the impact of ubiquitous computing have 
taken place while utilizing faculty and student survey results as well as Fact 
Book studies. The background of the strategic plan, assessment process and 
instruments, description of the computing initiative and outcomes of the 
ubiquitous computing program are described. 
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Assessment of the Impact of Ubiquitous Computing 

Background 

A Doctoral II private university is in the final stages of implementing an 
innovative strategic plan ("Plan for the Class of 2000," 1995) that has received attention 
from many sources and was featured in The Chronicle of Higher Education (DeLoughry, 
1995). Major elements of the plan include: 

• IBM laptop computers provided to all entering freshmen and faculty effective 
with the fall semester 1 996 

• A new first-year seminar course, ensuring each freshman an in-depth 
intellectual encounter effective with the fall semester 1996 

• 40 new tenure-track faculty members representing a 1 5 percent increase 

• Scholarships for 175 students to study abroad 

• Fellowships for 1 50 students to perform joint research with faculty members 

(Griffith, 1999) 

An evaluation committee consisting of faculty and administrators is conducting 
the assessment of the strategic plan. To encourage use of assessment data, the evaluation 
committee members serve as liaisons with different committees and administrative 
offices throughout the University. They provide the groups results of surveys and other 
data with analyses as appropriate (Griffith, 1999). 

The fundamental goals of the plan being measured by the evaluation committee 
are listed as follows: 

• Recommit to whole person — reason, faith and service 

• Strengthen student/faculty relations, especially one-on-one 

• Encourage greater student engagement with ideas 
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• Improve the effectiveness and efficiency of learning and teaching, inside and 
outside the classroom 

• Prepare students for the information age. 

• Increase recognition of the diversity of our present world in students, faculty 
and programs 

• Increase the quality/quantity of scholarship 

The evaluation committee also monitors twenty-seven key measures in the 
following areas: quality at the entrance door, quality of activities while students attend the 
school, quality at the exit door and opinions concerning quality (Griffith, 1999). 

"Examples of key measures being monitored are freshman retention rates, student/faculty 
ratio, percent of graduates receiving degree credits abroad, graduation rates, alumni giving 
rates and academic reputation as measured in U. S. News & World Report" (Griffith, 1999). 

Assessment Instruments 

The primary instruments being used to measure the effectiveness of the entire 
strategic plan are as follows: 

College Student Experiences Questionnaire (CSEQ) - The survey was 
administered to freshmen, sophomores and juniors 1996 - 1998. It is designed to indicate 
how students use their time. Comparative data with other schools are available while the 
institution's CSEQ results were highlighted in Change (Banta and Kuh, 1998). 

Cooperative Institutional Research Program (CIRP) Freshman Survey - The 
survey has been administered to the freshmen in different years since 1980. Trend 
analysis has enabled the school to learn about demographic changes of the entering 
freshmen. Comparative data with other schools are available. 
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Freshman Essay - The freshman classes of 1996, 1997 and 1998 were asked to 
respond to questions about their expectations for college. A follow-up essay 
questionnaire will be administered to the respondents upon their graduation to enable 
comparisons with original responses. 

Higher Education Data Sharing (HEDS) Consortium Alumni/ae Survey - The 

survey was administered to the classes of 1989 and 1993. Results enable the institution 
to assess the effects of the undergraduate education on graduates' lives five years later. 
Comparative data with peer institutions are provided by HEDS. 

HEDS Senior Survey - The survey was administered to the graduating classes of 
1993-1998. Results of the survey enable the school to assess graduating seniors’ overall 
experiences. Comparative data with peer institutions are also provided by HEDS. 

Higher Education Research Institute (HERI) Faculty Survey - The survey was 
administered to the undergraduate and MBA faculty in 1998. Comparative data of other 
institutions will be provided by HERI. 

In-house Faculty Computer Survey - The survey was developed by faculty in the 
department of communication to assess faculty use of computers. It has been administered to 
undergraduate faculty in 1996 - 1999. 

In-house Faculty Survey (1998) - The survey was developed by the evaluation 
committee to assess the effectiveness of all portions of the strategic plan and was 
administered to the undergraduate faculty in 1995 and 1998. 

In-house Student Computer Survey - Faculty in the department of 
communication developed the survey to assess students’ use of computers and how 
computers aid them educationally. It was administered to the students in 1996 - 1999. 
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Fact Book (1999) - This publication, produced annually since 1992, contains 
approximately 60 statistical studies about students, faculty and staff, facilities and finances. 

The Computing Initiative 

"The most distinctive outcome of the strategic plan is that all faculty and entering 
freshmen received IBM laptop computers effective fall 1996. They are given an upgrade 
every two years, and seniors keep the computers upon graduation" (Griffith, 1999). The 
faculty approved a standing committee on information technology, and this committee 
recommends, monitors and evaluates policies for computing in behalf of the faculty. The 
entire campus has been completely wired for computing, and seventeen professionals 
with the title of Academic Computing Specialist (ACS) have been employed to provide 
faculty support. A totally revamped Information Systems Support Center (ISSC) is open 
until midnight and provides immediate technical assistance to students, faculty and staff. 
The library staff provides a formal training program for both students and faculty in the 
use of computing. 

Faculty members have responded to these changes by establishing the "Computer- 
Enhanced Learning Initiative" (1998), known as CELL It is "faculty-based and has as its 
mission the development of effective uses of computers in instruction. Directed by a 
faculty member given a reduced teaching load, CELI sponsors campus seminars on using 
technology in instruction and provides a number of other services as well" (Griffith, 

1999). Also, Student Technology AdvisoRS (STARS) were created enabling students 
proficient in the use of computing to work in a collaborative role, one-on-one with 
faculty members. 
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Changes by Students and Faculty 

The implementation of ubiquitous computing of this magnitude is pioneering; 
fewer than twenty liberal arts colleges in the United States have such a program. Thus, 
many efforts have been made to assess its impact (Griffith, 1999). The CSEQ, In-house 
Faculty Survey, In-house Student Computer Survey and In-house Faculty Computer 
Survey were administered to students and faculty before and after the strategic plan was 
implemented. 

Utilizing .05 as the level of significance, the institution has been able to demonstrate 
changes of computer usage (McCoy, Griffith and Gu, 1999). CSEQ results show that 
significantly higher scores from students under the strategic plan took place in the 
following items: use of computers, courses using computers, use of computer in doing 
research and discussion of computers. See Figure 1 . 



Figure 1. College Students Experiences Questionnaire (CSEQ) 

Mean Responses 




Q1 Q2 Q3 Q4 Q5 Q6 



Q1 : Use of computers Q4: Computer training 

Q2: Courses use computers Q5: Discuss computers 

Q3: Computer skill Q6: Research; card catalogue & computer 

Faculty Survey results indicated increases for faculty under the plan in the following 

computer use items: teaching effectiveness, individual instruction, communication, 

presentation, modeling/simulation and information gathering. See Figure 2. 
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Figure 2. Faculty Survey Results 
Mean Responses 




Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q1 1 Q12 Q13 Q14 



Ql Computers in teaching 
Q2 Computers in communication 
Q3 Computers in individual instruction 
Q4 Computers for presentations 
Q5 Computers with information gathering 
Q6 Computers for modeling/simulation 
Q7 Computer skill 



Q8 Computer training & assistance 

Q9 Students proficient with computers 

Qi 0 Technology changed effectiveness of teaching 

Ql 1 Effect of computers on communication 

Q12 Effect of computers on resource material 

Q13 Effect of computers on presentations 

Q14 Use of technology in teaching 




Student computer skill has significantly increased, as reported by both students 
and faculty. Faculty perception of their own computer skill is significantly greater in 
1998 than it was in 1995. According to the CSEQ, student means in 1997 were 
significantly higher than that of 1996 in computer skill and computer training. See 
Figure 1. On the Faculty Survey, mean scores for computer skill and computer support 
and training are significantly higher in 1998 than in 1995. Faculty reported that a 
significantly higher number of their students were proficient in use of computers in 1998 
than in 1995. See Figure 2. 

Results from the In-house Student Computer Survey and In-house Faculty 
Computer Survey have revealed different types of computer behaviors. For example, 
students under the computing initiative use e-mail to communicate in general but prefer 
to communicate with faculty and other students face-to-face; faculty attitudes toward 
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computing overall have become more positive imder the computing initiative and the 
highly skilled faculty tend to use the computer for specialized purposes. (Mitra and 
Hazen, 1999) 

In non-computer areas, the CSEQ results (1997 vs. 1996) indicate that students 
under the computing initiative participated more in class discussions and had more 
positive relationships with faculty and administration. However, these students had 
significantly lower scores in items regarding discussion of music, economy and personal 
situations as well as lower participation in volunteer activities, attending performing arts 
and other related campus events. These results suggest, at least initially, that the 
initiative might have impacted students so that they may be less rounded than previous 
students. 

In non-computer areas for faculty, significant differences from the Faculty Survey 
indicated higher scores for intellectual climate among students as well as religious 
development of students and students being prepared for graduate/advanced education and 
employment after college. However, the faculty had significantly lower scores in number 
of papers or compositions published as well as number of professional meetings attended. 
These results suggest that computer training might have occupied time for faculty they 
otherwise would use in research and professional pursuits. 

Academic Outcomes 

There is indication that the computing initiative has had a positive impact on 
academic achievement. The retention rate of first-year students who entered in 1 996 (the 
first year of the strategic plan) was higher than that of those entering in the previous three 
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years. Also, the 94.3% retention rate of the 1997 freshman class was even higher than 
the 1996 freshman class retention rate of 93.4%. See Figure 3. 



Figure 3. Freshman Retention Rate 
Following Year 




The mean grade point average of first-year students at the end of the first year was higher 
in 1996 (2.81) and 1997 (2.83) than in the immediate years prior to the implementation of the 
strategic plan. See Figure 4. 



Figure 4. Mean Cumulative Grade Point Average 
End of Freshman Year 
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With the improvements indicated, the academic quality of the entering freshman 
class has been essentially unchanged as far as students' mean SAT scores and academic 
rankings in high school graduating classes (McCoy et al., 1999). 

Summary 

The data suggests that the computing initiative of the strategic plan has been 
effective to this date based upon the faculty and student survey results as well as retention 
and average GPA of freshmen who enrolled under the initiative (McCoy et al., 1999). 

The computing initiative received such a vast amount of interest from other institutions 
that the International Center for Computer Enhanced Learning (ICCEL) was formed 
(1997) at the institution, and faculty and administrators have been satisfied with the state 
of the strategic plan (Cox, 1998). 

However, recurring and thorough assessment is necessary since the first class to 
enroll under the computing initiative will not graduate until 2000. Senior and faculty 
survey results as well as measures including graduation rates and other outcomes can 
then be compared to previous classes and faculties and comparable institutions to 
evaluate the complete effectiveness of the computing initiative. 
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